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I.  Summary:  Objectives  and  Status  of  Effort 

In  this  report  we  summarize  our  accomplishments  under  the  research  program 
supported  by  Grant  F49620-00- 1-0362.  The  basic  scope  of  this  research  program  is  to 
carry  out  fundamental  research  in  several  interrelated  areas:  (a)  the  development  of 
methods  for  the  statistical  modeling  of  complex  phenomena  using  multiresolution, 
hierarchical,  and  relational  structures;  (b)  the  investigation  of  sensor  fusion  algorithms  for 
very  complex  space-time  phenomena  and  activities,  especially  exploiting  the  types  of 
structures  developed  in  the  first  research  area;  (c)  the  development  of  statistical  models 
for  shapes  and  their  use  in  developing  robust  and  statistically  optimal  methods  of  shape 
estimation  and  recognition;  and  (d)  the  investigation  of  methods  for  blending  physics  and 
methods  of  statistical  learning  in  order  to  devise  new  algorithms  for  image 
reconstruction,  feature  extraction,  and  fusion. 

Key  features  of  this  effort  are  that  (i)  it  blends  together  methods  from  several 
fields— statistics  and  probabilistic  modeling,  signal  and  image  processing,  mathematical 
physics,  scientific  computing,  Bayesian  networks,  statistical  learning  theory,  and 
differential  geometry-to  produce  new  approaches  to  emerging  and  challenging  problems 
in  signal  and  image  processing;  (ii)  it  both  builds  on  the  research  results  we  have 
obtained  under  our  current  grant  and  also  explores  new  directions  in  which  our 
approaches  appear  to  have  significant  merit;  and  (iii)  each  aspect  of  the  proposed 
program  contains  both  fundamental  research  in  mathematical  sciences  and  important 
applications  of  direct  relevance  to  Air  Force  missions.  In  particular,  our  research  is 
directly  relevant  to  problems  including  automatic  target  recognition  based  on  synthetic 
aperture  radar  (SAR)  and  laser  radar  imagery,  wide-area  surveillance  and  information 
preparation  of  the  battlefield  (as  envisioned,  for  example,  in  the  “Targets  under  Trees” 
initiative  formulated  at  the  request  of  the  Air  Force  Chief  of  Staff);  global  awareness  and 
higher-level  fusion  for  situational  assessment;  and  fusion  of  multiple  heterogeneous 
sensors  as  required  to  realize  the  vision  for  Predictive  Battlespace  Awareness  and 
“Cursors  on  Target”  concepts  defined  by  the  CSAF  and  recently  examined  in  detail  by 
the  USAF  Scientific  Advisory  Board.  In  all  of  these  areas  we  have  direct  and  strong 
contacts  and  interactions  with  AFRL  staff  and  with  industry  involved  in  Air  Force 

programs. 

/ 

The  principal  investigator  for  this  effort  is  Professor  Alan  S.  Willsky.  Prof. 
Willsky  is  assisted  in  the  conduct  of  this  research  by  Dr.  John  Fisher,  research  scientist  in 
Prof.  Whisky's  group,  by  Dr.  Mujdat  Cetin,  postdoctoral  researcher  in  Prof.  Willsky’ s 
group  and  by  several  graduate  research  assistants  as  well  as  additional  thesis  students  not 
requiring  stipend  or  tuition  support  from  this  grant.  In  the  next  section  we  describe  our 
research  accomplishments;  in  Section  m  we  indicate  the  individuals  involved  in  this 
effort;  in  Section  IV  we  list  the  publications  supported  by  this  effort;  and  in  Section  V  we 
discuss  interactions  and  transitions. 
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II.  Accomplishments 

In  this  section  we  briefly  describe  the  research  accomplishments  we  have 
achieved  with  support  provided  by  this  grant.  We  limit  ourselves  here  to  a  succinct 
summary  and  refer  to  the  publications  listed  at  the  end  of  this  report  for  detailed 
developments.  However,  we  do  note  here  that  our  work  continues  to  have  significant 
impact,  both  in  terms  of  DoD-related  activities  and  transitions  in  progress  (Section  V) 
and  in  terms  of  recognition  from  the  research  community  (including  the  upcoming  (Sept. 
2003)  plenary  lecture  to  be  delivered  by  Prof.  Willsky  at  the  IEEE  Workshop  on 
Statistical  Signal  Processing). 

2.1  Multiresolution,  Hierarchical,  and  Relational  Modeling 

The  research  described  in  this  section  is  developed  in  great  detail  in  a  number  of 
papers  and  reports  [1,4,5,13,19-21,23-26,32-35, 47, 50-51, 55,  58-62,  64-66,  69,  71-72, 
74,  77,  82,  85-87,  92-94].  The  overall  objective  of  this  portion  of  our  research  is  the 
development  of  methods  for  constructing  stochastic  models  for  phenomena  that  vary  over 
space,  time,  and  hierarchy  and  that  possess  structure  which  can  be  exploited  to  construct 
efficient  and  scaleable  algorithms  for  statistical  inference  (the  subject  of  subsequent 
sections  of  this  report). 
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a)  We  have  continued  our  research  on  construction  of  statistical  models  on 
multiresolution  trees.  Our  most  recent  results  involve  a  new  and  much  tighter 
characterization  of  the  conditional  independencies  that  must  hold  among  the 
values  of  a  stochastic  process  defined  on  such  a  tree  so  that  that  process  is 
Markov  (more  generally  we  develop  conditions  for  Markovianity  when  we  wish 
to  match  the  distribution  of  the  process  values  at  a  subset  of  the  nodes  of  the  tree). 
This  new  characterization  leads  directly  to  new  algorithms  for  construction  of 
exact  and  approximate  multiresolution  models  starting  either  from  the  covariance 
specification  of  the  “target”  process  (or  more  precisely  the  values  of  the  process 
on  the  set  of  “target”  nodes  whose  joint  distribution  we  wish  to  model  accurately) 
or  directly  from  data. 

b)  We  are  also  continuing  our  work  on  building  models  for  certain  types  of  non- 
chordal  graphs  with  loops  (i.e.,  graphs  for  which  there  is  not  necessarily  a  simple 
aggregation  of  nodes  to  produce  trees).  The  motivation  for  this  work  comes  from 
large-scale  remote  sensing  problems  such  as  arise  in  oceanographic  data 
assimilation,  in  which  we  would  expect  to  have  both  fine-scale  covariance 
information  locally  but  coarser-scale  covariance  information  over  larger  areas — 
e.g.,  typical  specifications  of  ocean  statistics  will  specify  covariances  of  processes 
at  different  resolutions,  corresponding  to  specifying  the  covariance  between 
spatial  averages  of  the  field.  Such  specifications  lead  to  non-chordal  graphs  in 
which  the  coarser-scale  variables  whose  correlations  are  known  reside  at  coarser 
scales  in  a  multiresolution  graph.  We  are  currently  exploring  application  and 
extension  of  some  recent  results  on  non-chordal  covariance  extension  and 
semidefinite  programming  to  solve  problems  of  this  type. 
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c)  A  broad  area  for  the  extension  of  our  multiresolution  methodologies  is  to  non- 
Gaussian  and  nonlinear  models  on  both  trees  and  on  graphs  with  loops.  Our  most 
recent  work  in  this  area,  which  represents  a  new  direction  for  us,  is  the  use  of 
nonparametric  density  estimation  techniques  to  compute  estimates  of  distributions 
at  individual  nodes  and  joint  distributions  between  pairs  of  nodes.  See  the 
discussion  in  the  next  section  for  more  on  this  approach,  which  simultaneously 
solves  problems  in  modeling  and  in  fusion. 

d)  We  have  continued  our  research  on  a  new  approximate  MR  modeling  framework, 
which  we  refer  to  as  Recursive  Cavity  Modeling  that  works  directly  with  the 
inverse  of  the  covariance  of  the  process  to  be  modeled.  In  particular,  for  an  NxN 
covariance,  if  we  reconstruct  a  graph  with  nodes  { 1 , . . .  ,N}  and  an  edge  {i  j }  if  the 
ijth  element  of  the  inverse  of  the  covariance  matrix  is  nonzero,  then  the  process  is 
Markov  with  respect  to  this  graph.  An  important  property  of  such  a  model  is  that, 
conditioned  on  the  values  of  the  process  on  any  separator  set  of  nodes  that 
partitions  the  graph  into  two  or  more  disconnected  components,  the  sets  of  values 
of  the  process  on  these  disconnected  components  are  mutually  independent.  This 
leads  to  a  general  methodology  for  constructing  multiresolution  models 
corresponding  to  a  multiresolution  sequence  of  nested  partitions  of  this  type,  with 
very  fine  partitions  at  finer  scales  and  coarser  partitions  at  coarser  scales.  The 
problem  with  this  method  is  that  the  dimensionality  of  the  variables  that  form  the 
state  at  any  node  of  such  a  model  is  proportional  to  the  number  of  nodes  in  the 
separator  set  corresponding  to  that  node,  and  at  coarser  scales  in  the  model,  that 
dimensionality  can  be  extremely  large.  Thus  direct  specification,  manipulation, 
and  storage  of  the  covariance  of  such  a  state  is  prohibitively  complex.  However, 
while  it  is  generally  the  case  that  the  inverse  of  these  state  covariances  are  full,  for 
many  problems  they  are  well  approximated  by  sparse  matrices,  corresponding  to 
Markov  models  for  each  of  these  separator  sets.  Our  approach  involves 
developing  methods  for  the  direct  propagation  of  these  sparse  approximate 
models,  using  the  same  two-sweep  (fine-to-coarse,  coarse-to-fine)  algorithmic 
structure  as  with  our  other  multiresolution  algorithms.  Since  each  of  these 
boundaries  encloses  a  region  of  the  graph,  we  refer  to  these  as  cavity  models. 
During  this  past  year  we  have  developed  a  very  general  framework  that  makes 
extensive  use  of  information-geometry  to  develop  iterative  algorithms  for  optimal 
recursive  cavity  modeling  applicable  to  general  graphical  models — i.e.,  not 
limited  to  Gaussian  processes  and,  in  fact,  just  as  applicable  to  nonlinear,  discrete, 
and  hybrid  continuous/discrete  data.  We  are  currently  working  on  a  first  paper 
developing  and  applying  the  method  for  the  Gaussian  processes. 

e)  As  discussed  in  the  next  subsection,  we  have  had  great  success  with  the 
development  of  a  new  class  of  algorithms  that  we  refer  to  as  Embedded  Tree  (ET) 
Algorithms.  The  success  of  these  algorithms  has  motivated  the  initiation  of 
research  on  building  multiresolution  models  on  structures  for  which  ET 
algorithms  offer  superior  performance.  Among  these  structures  are  models  on 
what  we  refer  to  as  “near-tree”  structures,  i.e.,  multiresolution  trees  augmented 
with  a  comparatively  small  number  of  “cross-links”  between  nodes  that  are  close 
spatially  but  much  more  distant  as  measured  along  the  graph  prior  to 
augmentation.  These  models  allow  us  to  overcome  one  of  the  limitations  of  pure 
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tree  models,  namely  the  potential  for  errors  and  artifacts  across  such  major  tree 
boundaries,  while  the  ET  algorithm  allows  us  to  exploit  the  efficiency  of  tree- 
based  models.  We  have  now  demonstrated  that  such  models,  with  the 
introduction  of  very  few  edges,  can  lead  to  models  with  dramatically  superior 
approximation  power  (and  with  little  loss  in  the  efficiency  of  associated 
estimation  algorithms).  We  are  currently  working  on  efficient  methods  for 
building  such  models  by  adding  edges  successively,  in  part  borrowing  from  the 
same  set  of  tools  used  in  our  work  on  recursive  cavity  modeling. 


2.2  Sensor  Fusion  Algorithms  over  Space,  Time,  and  Hierarchy 

The  research  described  in  this  section  deals  with  efficient  algorithms  for  large- 
scale  optimal  estimation  and  is  reported  in  detail  in  [3,6,20,23,26,30-35, 47,  50-51,  55, 
58-62, 64-66, 69,  71-72,  74, 77,  82,  85-87, 92-94].  The  general  objective  for  this  part  of 
our  research  is  to  investigate  stochastic  models  with  structure  that  can  be  exploited  in 
order  to  develop  optimal  or  near-optimal  algorithms  that  are  scaleable  to  the  large-scale 
problems  encountered  in  image  analysis,  sensor  fusion,  and  higher-level  fusion.  Our 
research  here  involves  the  development  of  algorithms  that  parallel  the  modeling  efforts 
described  in  the  preceding  subsection  as  well  as  several  other  initiatives  involving 
graphical  modeling  concepts: 

a)  As  mentioned  in  the  preceding  subsection,  we  also  have  had  considerable 
success  in  our  development  of  Embedded  Tree  (ET)  Algorithms.  The 
objective  of  this  work  is  to  exploit  our  powerful  algorithms  for  estimation  and 
fusion  on  trees  in  order  to  develop  more  global  algorithms  than  the 
predominant  algorithmic  concept  in  the  field  of  graphical  models,  namely 
Pearl’s  algorithm,  which  is  optimal  for  trees  but  provably  suboptimal  for 
graphs  with  cycles  (in  particular  for  linear-Gaussian  problems,  Pearl’s 
algorithm,  when  it  converges,  gives  the  correct  estimates  but  incorrect  error 
covariances).  In  our  algorithmic  framework,  we  view  the  edge  set  of  a  graph 
with  loops  as  the  union  of  edge  sets  of  spanning  trees  each  of  which  comes 
from  discarding  a  subset  of  the  edges  of  the  original  graph.  Using  this 
structure  we  can  directly  define  iterative  algorithm  in  which  we  perform  tree- 
based  estimation  using  each  of  these  embedded  tree  models  in  succession. 

We  have  shown  that  such  algorithms,  when  they  converge,  both  yield  the  . 
optimal  estimates  and  the  correct  error  covariances.  Furthermore, 
experimental  results  demonstrate  that  these  algorithms  converge  in  many 
more  situations  than  Pearl’s  algorithm.  Indeed,  by  exploring  ties  of  these 
methods  to  work  in  numerical  linear  algebra  for  the  solution  of  large  linear 
systems,  we  have  both  developed  new  algorithms  for  that  community  plus 
borrowed  some  ideas  to  develop  even  more  powerful  algorithms  for  graphical 
model  estimation  that  result  from  using  our  embedded  tree  algorithm  as  a 
preconditioner  for  conjugate  gradient  algorithms.  We  have  demonstrated  the 
power  of  these  methods  for  large-scale  multisensor  fusion  problems 
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(involving  thousands  of  sensing  nodes)  and  in  estimation  of  random  fields  and 
processes. 

b)  As  was  also  discussed  in  the  preceding  section,  we  have  been  working  on  the 
development  of  a  methodology  for  Recursive  Cavity  Modeling  and 
Estimation.  In  particular,  the  methodology  for  constructing  these  models 
directly  leads  to  efficient  two-sweep  algorithms  for  estimation  and  fusion, 
propagating  information  around  the  boundary  of  each  cavity  and  then 
propagating  first  outward  as  cavities  grow  and  are  merged  and  inward.  We 
have  demonstrated  the  power  of  these  algorithms  for  a  variety  of  large  Gauss- 
Markov  random  fields,  and  a  paper  on  these  results  is  in  progress.  Moreover, 
as  mentioned  in  the  preceding  section,  we  have  extended  the  concept  of 
recursive  cavity  modeling  to  general,  non-Gaussian  graphical  models,  using 
powerful  ideas  and  results  from  information  geometry,  and  we  are  now 
beginning  to  investigate  the  utility  of  this  extension  to  a  variety  of  estimation 
and  distributed  fusion  problems. 

c)  We  have  continued  our  work  on  several  different  algorithms  and  problems 
related  to  what  we  have  called  tree-reparametrization  (TRP)  algorithms.  The 
work  reported  previously — and  on  which  we  continue  to  work  as  well — 
focuses  on  the  computation  of  marginal  distributions  at  each  node  in  a 
graphical  model — the  same  problem  for  which  Pearl’s  belief  propagation 
algorithm  (alternately  known  as  the  sum-product  algorithm)  represents  an 
approximate  solution.  Each  iteration  of  a  TRP  algorithm  involves  operations 
over  a  tree  embedded  in  the  graph.  The  critical  idea  here  is  the  recognition 
that  optimal  algorithms  on  trees  correspond  to  performing  a  refactorization  of 
the  probability  distribution  for  the  entire  process,  one  that  explicitly  exposes 
the  marginal  distributions  for  each  node  in  the  graph.  TRP  algorithms 
iterative  perform  this  refactorization  over  a  set  of  trees.  In  our  work  we  have 
demonstrated  that  this  algorithm  has  better  convergence  properties  than 
Pearl’s  algorithm  and  have  also  developed  important  theoretical  results  on  the 
characterization  of  fixed  points  of  these  iterations,  on  necessary  conditions  for 
convergence  using  a  pair  of  spanning  trees,  and  bn  bounds  on  the  errors  in 
these  algorithms.  The  latter  results  involve  careful  use  of  concepts  in  convex 
duality  to  obtain  methods  for  optimizing  our  bounds  over  all  embedded  trees 
in  a  graph.  Since  there  are  generally  many  embedded  trees,  performing  this 
optimization  directly  is  completely  intractable.  However,  the  use  of  a  dual 
formulation  reduces  this  to  a  remarkably  simple  optimization  problem.  More 
recently  we  have  focused  on  another  problem  of  great  practical  importance — 
e.g.,  in  multisensor  data  association  and  in  coding  applications — namely  that 
of  computing  the  overall  MAP  estimate,  i.e.,  the  peak  of  the  overall  joint 
distribution  for  all  variables  on  the  entire  graph.  If  the  graph  is  a  tree — i.e., 
contains  no  cycles — this  computation  can  be  performed  very  efficiently  either 
in  a  two-sweep  fashion,  generalizing  the  dynamic  programming  structure  of 
the  celebrated  Viterbi  algorithm  or  in  a  local  message-passing  algorithm, 
analogous  to  the  sum-product  algorithm  and  often  referred  to  as  the  max- 
product  algorithm.  However,  if  the  graph  of  interest  contains  loops, 
performing  MAP  estimation  is,  in  general  NP-Hard,  and  the  application  of  the 
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max-product  algorithm  to  such  graphs  not  only  may  not  converge  but  also 
leads  to  suboptimal  solutions  when  convergence  does  occur.  In  our  work  we 
have  developed  counterparts  of  our  TRP  algorithms  that  are  aimed  at  the  MAP 
problem  instead.  In  part  this  work  allows  us  to  develop  both  algorithms  that 
converge  more  frequently  than  standard  max-product  algorithms  and  also 
analyses  of  the  properties  of  fixed  points.  In  addition,  by  adapting  our  ideas 
on  using  multiple  trees,  we  have  been  able  to  develop  a  tree-reweighting 
algorithm  that  is  guaranteed  to  converge  to  the  true  MAP  estimate  under  fairly 
mild  conditions.  Indeed  these  conditions  in  essence  define  the  space  of 
problems  in  which  distributed  message-passing  algorithms  such  as  max- 
product  or  our  new  ones  should  be  applied  (as  when  these  conditions  are 
violated,  MAP  estimation  cannot  be  performed  solely  via  local  computations). 

d)  One  of  the  motivations  for  the  work  just  described  on  MAP  estimation  is  the 
development  of  efficient  algorithms  for  multisensor,  multitarget  data 
association,  a  notoriously  complex  problem.  We  have  now  demonstrated  that 
our  new  approach  to  max-product  and  tree-reweighting  algorithms  can  yield 
remarkably  efficient  solutions  to  optimal  data  association  problems  that  have 
heretofore  been  considered  too  complex  for  practical  solution  (hence  requiring 
the  use  of  heuristics  to  obtain  tractable,  but  suboptimal,  solutions).  This  work 
is  now  focused  on  several  extensions,  including  to  the  examination  of  how 
convergence  varies  as  a  function  of  the  set  of  trees  used  (ranging  from  “trees” 
consisting  of  single  pairs  of  nodes  as  in  the  max-product  algorithm  to  full 
spanning  trees),  number  of  iterations,  and  the  difficulty  of  the  association 
problem  as  measured  in  terms  of  the  level  of  contention  (multiple 
measurements  contending  for  association  with  particular  targets  and  multiple 
targets  contending  for  association  with  particular  measurements). 

e)  As  mentioned  in  the  previous  section,  we  are  developing  new  approaches  to 
inference  for  graphical  models  that  involve  non-Gaussian  densities — problems 
of  particular  importance  for  various  sensing  modalities  that  provide 
measurements  of  either  bearing  or  range.  These  methods,  which  involve  the 
use  of  methods  for  nonparametric  density  estimation  (for  which  reason  we 
refer  to  them  as  Nonparametric  Belief  Propagation  algorithms),  can  be 
viewed  as  extensions  of  concepts  of  particle  filtering  to  inference  on  graphs — 
this  extension  is  highly  nontrivial,  especially  for  graphs  with  loops,  as  the 
iterative  computations  and  generation  of  messages  of  belief  propagation 
require  new  ideas  for  generating  “particles”  to  replace  those  messages.  In 
addition  to  developing  the  basic  methodology,  we  are  also  exploring 
applications  in  both  computer  vision  and  in  fusion  for  sensor  networks. 

f)  Finally,  motivated  by  problems  of  fusion  for  sensor  networks,  we  have 
recently  initiated  a  new  look  at  message-passing  algorithms  for  inference  on 
graphs,  namely  a  look  that  brings  in  both  the  cost  of  communication  as  well  as 
ideas  from  other  disciplines,  most  notably  decentralized  estimation  and 
hypothesis  testing.  Our  intention  here  is  to  develop  new  approaches  that 
synthesize  concepts  from  this  disparate  disciplines.  Among  the  problems 
being  investigated  are  (1)  including  additional  protocol  bits  in  messages  in 
belief  propagation  or  TRP  algorithms  that  provide  information  on  the  path  that 
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each  message  has  taken  and  hence  allow  receiving  sensor  nodes  to  account  for 
the  fact  that  information  embedded  in  these  messages  maybe  correlated 
(because  of  loops  in  the  network  graph);  (2)  examining  problems  of  finite-bit 
messages  in  belief  propagation,  generalizing  ideas  of  decentralized  detection 
to  loopy  graphs;  and  (3)  building  on  results  in  decentralized  detection  to 
develop  algorithms  for  network  self-organization,  including  determining  how 
to  distribute  estimation  responsibility  to  nodes  in  a  network  (i.e.,  which  node 
estimates  which  variables). 


2.3  Statistical  Modeling  and  Estimation  of  Shape  with  Applications  in  Object 
Extraction  and  Recognition 

The  general  objective  of  this  part  of  our  research  is  the  development  of 
statistically  robust  methods  for  segmentation  and  shape  estimation  with  applications 
ranging  from  wide-area  mapping  to  object  recognition,  and  the  results  of  our  efforts  in 
this  area  are  developed  in  detail  in  publications  [10-12,22,36-41, 46,  54,  57,  74, 78-84, 

88,  91, 102-103].  Essentially  all  of  our  recent  work  in  this  area  has  focused  on  so-called 
curve  evolution  methods  and,  in  particular,  on  developing  statistically-based  curve 
evolution  algorithms.  Our  earlier  work — e.g.,  on  developing  curve  evolution  algorithms 
for  the  image  processing  problem  of  Mumford  and  Shah — has  received  considerable 
recognition,  motivating  the  work  we  now  describe: 

a)  In  recent  research  we  have  extended  our  curve  evolution  ideas  to  allow  us  to 
include  blurring — i.e.,  to  consider  image  deblurring  as  well  as  denoising  and 
segmentation — as  part  of  a  single,  unified  formulation.  Interestingly,  in  this 
case  the  PDE’s  inside  and  outside  the  curve  are  now  coupled,  thanks  to  the 
blurring  of  the  image  data,  and  there  are  a  number  of  interesting  and  important 
theoretical  and  algorithmic  issues  to  which  this  gives  rise  and  which  are 
currently  under  investigation. 

b)  One  of  the  major  areas  of  our  current  and  future  research  in  this  area  is  that  of 
incorporating  prior  information  about  shape  into  curve  evolutions.  This  is 
particularly  important  for  problems  in  which  image  SNR  is  low  or  in  which 
the  objects  of  interest  are  partially  occluded.  Major  issues  here  include  the 
development  of  methods  for  constructing  prior  probability  distributions  on 
shapes  from  examples  and  the  incorporation  of  these  priors  into  curve 
evoloution  formalisms.  In  our  initial  work  in  this  area  we  used  a  set  of 
training  examples  to  construct  a  set  of  “eigenshapes,”  which  then  are  used  to 
provide  a  linear  parameterization  of  a  set  of  shapes,  where  the  parameters  of 
that  linear  parametrization  is  then  estimated  as  part  of  the  curve  evolution 
process.  Results  on  both  military  and  medical  images  in  both  2-D  and  3-D 
have  demonstrated  that  this  methodology  has  a  great  deal  of  promise.  A  paper 
on  this  topic  received  the  Best  Paper  Award  at  IEEE  CVPR,  and  recently  we 
have  extended  these  results  to  problems  in  which  there  are  multiple  objects  to 
be  segmented  simultaneously.  A  key  idea  here  is  that  by  doing  this,  we  can 
use  knowledge  of  the  relative  locations  and  shapes  of  a  set  of  objects  to  allow 
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us  to  couple  the  estimation  of  each.  For  example,  if  one  object  is  particularly 
easy  to  segment  in  an  image,  the  availability  of  a  joint  model  allows  us  to  use 
this  segmentation  to  guide  the  segmentation  of  other  objects  that  may  be  more 
difficult  to  segment  on  their  own. 

c)  We  have  also  continued  our  work  on  an  approach  that  blends  both  curve 
evolution  methods  as  described  in  this  subsection  and  the  nonparametric 
methods  for  statistical  learning  discussed  in  the  next.  In  particular,  we  have 
developed  a  method  for  image  segmentation  that  assumes  only  that  the  image 
under  question  consists  of  two  types  of  regions,  each  of  which  is  characterized 
by  a  different  but  completely  unknown  probability  distribution.  The  approach 
we  have  taken  is  that  of  attempting  to  estimate  these  probability  distributions 
at  the  same  time  as  we  are  performing  segmentation.  The  approach  which 
(for  large  images)  is  asymptotically  equivalent  to  full  maximum  likelihood 
segmentation,  involves  explicitly  nonparametric  estimation  of  the  probability 
distributions  given  a  current  estimated  segmentation  together  with  a  curve 
evolution  that  attempts  to  increase  the  Kullback-Leibler  divergence  between 
the  estimated  distributions.  The  curve  evolution  actually  involves  several 
other  terms  which  account  for  the  fact  that  we  may  be  dealing  with  small 
images  or,  more  importantly,  with  the  detection  of  relatively  small  regions 
within  an  image.  In  this  case,  the  curve  evolution  must  account  for  the  fact 
that  evolving  the  curve  will  change  the  set  of  pixels  considred  to  be  within  a 
given  region  which  in  turn  will  change  the  estimated  probability  distribution. 
An  invited  paper  on  this  work  will  be  presented  at  the  upcoming  ICIP 
conference. 

d)  Finally,  we  have  recently  initiated  a  new  effort  aimed  at  space-time  tracking 
of  curves  or  boundaries  that  evolve  in  time  themselves.  This  is  of  particular 
importance  in  problems  such  as  estimating  local  variations  in  meteorological 
conditions  in,  say,  a  battlefield  environment  (as  envisioned  in  the  so-called 
WeatherWeb  concept).  Our  approach  to  this  problem  involves  developing 
and  exploiting  models  for  the  probabilistic  evolution  of  such  curves — in 
essence  developing  temporal  Markov  models  for  these  curves.  We  are 
exploring  several  approaches  to  this  problem,  including  one  that  blends  the 
learning  of  linear  parameterizations  of  curves  as  described  in  item  “b”  above 
with  our  methods  for  learning  dynamic  models  in  time,  as  described  in  the 
following  section. 


2.4  Blending  Physics  and  Statistical  Learning  for  Image  Reconstruction,  Feature 
Extraction,  and  Fusion 

In  this  section  we  describe  our  research  on  marrying  sensor  physics  with  statistics 
and  nonparametric  statistical  methods  in  order  to  develop  robust  methods  for  exploiting 
the  information  present  in  sensor  data.  The  fundamental  idea  here  is  that  a  full,  heavy  use 
of  fundamental  physics  (e.g.,  solving  Maxwell’s  equations)  is  clearly  inappropriate  for 
sensor  fusion  since  there  are  limitations  in  the  “apertures”  through  which  we  view  the 
phenomenon  both  from  the  input  and  output  sides.  In  particular,  the  data  that  are 
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typically  available  (e.g.,  to  form  a  SAR  image)  are  far  too  limited  in  extent  and  subject  to 
too  many  sources  of  uncertainty  and  variability  to  warrant  full  solution  of  the  inverse 
scattering  problem  in  order  to  form  an  image.  Fortunately,  and  complementary  to  the 
limitations  in  the  observed  data  is  the  fact  that  the  desired  information  we  wish  to  extract 
from  such  sensor  data-e.g.,  detections  and  classifications  of  objects--are  far  more  limited 
than  a  complete  inverse  scattering  solution.  On  the  other  hand,  discarding  all  physics 
also  is  unwise,  as  the  constraints  implied  by  sensor  physics  can  be  used  to  reduce  the 
apparent  number  of  degrees  of  freedom  in  the  data,  thus  enhancing  the  quality  of  any 
statistical  analysis.  The  challenge  is  to  determine  the  proper  balance  between 
physics/model-based  methods  and  statistical/leaming  methods.  This  is  a  deep  and 
enduring  theme  to  which  we  believe  we  have  made  some  contributions.  Our  work  to  date 
in  this  broad  area  is  described  in  [2,7-9,14-18,27-29,42-45, 48-49,  52-53,  56,  63,  67-70, 
73,  76,  83,  89,  90, 95-101, 104-108]. 

a)  In  this  part  of  our  research  we  have  taken  a  deeper  look  at  marrying  SAR 
physics  with  nonparametric  statistical  learning  methods  for  constructing 
probabilistic  models  for  multiresolution  imagery.  In  particular  consider  the 
formation  of  SAR  imagery  based  on  a  given  full  aperture  of  data.  If  we  use 
the  entire  aperture,  we  obtain  imagery  at  the  finest  resolution  resolvable  using 
that  data.  However,  to  do  this  we  in  essence  must  assume  that  all  scattering  is 
isotropic,  i.e.,  that  the  response  from  significant  scatterers  is  constant  across 
the  entire  aperture.  For  many  important  scattering  mechanisms  this  is  not  the 
case  at  all,  and  this  anisotropy  is  critical  to  distinguishing  one  scatterer  type 
from  another.  Suppose  then,  that  in  addition  to  forming  an  image  using  the 
entire  aperture,  we  also  form  three  images  each  using  half  of  the  aperture:  one 
image  using  the  right  half,  one  the  left,  and  one  using  a  centered  half-aperture. 
If  indeed  there  are  anisotropic  scatterers,  we  might  expect  that  there  would  be 
differences  in  the  responses  in  each  of  these  half-apertures  and  hence  in  the 
images  formed  using  them  (note  that  these  images  would  have  pixel  sizes 
twice  as  large  as  the  ones  in  the  finest  scale  imagery).  Iterating  this  process, 
we  can  imagine  forming  a  vector  of  images  at  each  of  a  sequence  of  scales 
corresponding  to  progressively  smaller  subapertures.  By  looking  across  scale, 
then,  we  would  expect  not  only  to  find  statistical  variability  due  to  speckle  but 
also  any  evidence  of  anisotropic  scattering  manifesting  itself  in  statistically 
significant  differences  in  pixel  intensities  in  images  formed  using  different 
subapertures.  During  the  past  year  we  have  developed  hybrid  estimation 
algorithms  based  on  these  ideas.  Our  basic  algorithm  involves  both  the 
estimation  of  the  level  of  anisotropy  for  each  pixel  as  well  an  estimate  of 
reflectivity  consistent  with  the  anisotropy  estimate.  Together  these  provide 
both  sharpened  estimates  of  reflectivity  as  well  as  augmented  features  (namely 
anisotropy  designations)  that  should  be  useful  for  target  recognition.  One  of 
the  challenges  in  developing  such  an  algorithm  is  that  smaller  subapertures  do 
not  simply  measure  information  from  the  scatterer  pixel  of  interest  but  also  are 
corrupted  by  energy  from  neighboring  scatterers.  Thus,  our  hybrid  estimation 
procedure  for  estimating  anisotropy  and  reflectivity  for  a  given  pixel  must 
deal  with  “nuisance”  parameters  corresponding  to  scattering  from  neighboring 
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pixels.  In  our  basic  non-iterative  algorithm  this  is  accomplished 
independently  at  each  pixel,  in  which  effects  of  neighbors  are  estimated  and 
removed.  However,  a  more  sophisticated  approach  would  be  to  use  an 
iterative  procedure  in  which  the  actual  estimates  produced  at  neighboring 
pixels  at  each  iteration  were  used  to  enhance  the  next  iteration’s  estimate  of 
each  pixel.  Such  a  method  has  been  developed  and  promising,  enhanced 
imaging  results  have  been  demonstrated  in  which  spatially  extended  (and  thus 
highly  anisotropic)  scatterers  are  identified  and  dealt  with  in  a  way  that 
enhances  the  resulting  image  appearance. 

b)  A  continuing  and  very  active  component  of  our  research  focuses  on 
variational  methods  to  produce  enhanced  images  and  reconstructions  for  SAR, 
ISAR,  and  more  general  array  processing  applications.  In  particular,  by 
putting  particular  penalties  (e.g.,  Lp,  with  p  <  1)  either  on  the  reconstructed 
image  or  on  the  gradient  of  the  reconstructed  image,  we  have  shown  that  we 
can  produce  remarkably  sharp  images  of  point  scatterers  or  regions  and  can 
also  correct  for  phase  errors  due  to  target  motion — an  extremely  important 
problem  in  SAR  imaging  of  moving  targets  or  to  other  sources  (including 
timing  errors  to  array  element  location  errors).  Moreover,  in  contrast  to  many 
other  superresolution  methods  (e.g.,  MUSIC,  Capon’s  method),  our  method 
can  resolve  multiple  scattering  effects  that  are  highly  correlated — e.g.,  due  to 
the  presence  of  multipath  effects.  In  our  most  recent  efforts  we  have 
developed  new  variational  approaches  for  array  processing  that  work  well  for 
broadband  sources  and,  in  particular,  for  sources  that  generate  multiple 
harmonics  (e.g.,  as  are  present  in  any  motor  or  machinery). 

c)  We  have  also  continued  our  work  on  exploiting  ideas  from  nonparametric 
statistics,  information  theory,  and  machine  learning  to  develop  algorithms  for 
a  number  of  different  problems  in  signal  analysis  and  fusion.  One  such 
application  is  the  construction  of  dynamic  models  from  complex  signals  in  an 
unsupervised  learning  context.  The  principle  we  have  adopted  in  this  and  in 
our  other  work  in  this  area  is  that  of  maximizing  mutual  information.  In 
particular,  in  this  context,  the  objective  is  to  identify  functionals  of  the  past  of 
a  signal  that  have  maximal  mutual  information  with  the  next  value  of  the 
signal.  In  the  process  of  performing  that  optimization  using  nonparametric 
statistical  methods,  we  also  build  a  model  for  the  transition  probability  for  the 
process,  i.e.,  the  conditional  pdf  for  the  next  value  of  the  signal  given  these 
maximally  informative  functionals  of  the  past.  This  pdf,  then,  serves  as  a 
dynamic  model  of  the  phenomenon  which  can  be  used  for  signal  prediction, 
simulation,  discrimination,  and  estimation.  To  illustrate  the  power  of  this 
method,  we  have  applied  it  to  a  variety  of  signals  including  the  2-D  trajectory 
corresponding  to  signatures,  which  we  have  then  used  to  detect  forgeries.  We 
are  also  investigating  the  use  of  this  methodology  both  for  the  development  of 
statistical  models  for  dynamically  evolving  curves  (as  described  in  the 
preceding  section)  and  for  the  problem  of  separating  multiple  unknown 
sources  from  multiple  sensor  outputs. 
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d)  A  second  context  in  which  we  are  developing  nonparametric/information- 
theoretic  methods  is  that  of  the  fusion  of  signals  of  very  different  modalities. 

In  particular,  our  initial  work  in  this  area  has  focused  on  the  fusion  of  video 
imagery  and  acoustic  signals.  The  challenge  in  this  case  is  to  determine  what 
actions  in  the  video  sequence  are  responsible  for  what  components  in  the 
observed  acoustic  signals.  The  potential  applications  of  this  for  multisensor 
fusion  and  for  enhanced  source  separation  (e.g.,  using  video  to  enhance 
acoustic  signal  separation  or  using  audio  to  help  identify  significant  objects 
and  activities  in  image  sequences)  are  myriad.  The  results  on  preliminary 
imagery  are  striking  in  their  ability  to  localize  sources  of  sound  and  to  use 
video  to  guide  the  separation  of  acoustic  signals.  In  addition,  we  are  actively 
involved  in  developing  analogous  methods  to  provide  nonparametric  methods 
for  multisensor  data  association.  For  example,  in  complex  environments  it 
may  not  be  possible  to  combine  data  from  distributed  sensors  in 
straightforward  coherent  fashion,  e.g.,  due  to  variabilities  in  received 
signatures  from  different  viewing  angles  or  to  dispersive  or  nonlinear  effects 
that  disrupt  signal  coherency.  We  have  now  written  a  first  paper  on  this 
problem  and  are  working  on  a  general  approach  that  overcomes  the 
computational  complexities  that  arise  in  large-scale  data  association  problems. 
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V.  INTERACTIONS/TRANSITIONS 

In  this  section  we  summarize  the  various  interactions  and  transitions  associated 
with  research  supported  by  AFOSR  Grant  F49620-00-0362. 

Participation/Presentation  at  Meetings 

In  addition  to  presentations  at  professional  conferences,  we  have  been  involved  in 
the  following  other  meetings: 

(1)  Prof.  Willsky  participated  in  the  joint  DUST/D  ARP  A  workshop  on  future 
directions  for  ATR,  held  in  July  2001 .  This  meeting,  attended  by  invitation 
only  primarily  by  government  and  industry,  had  as  its  aim  the  identification  of 
the  next  generation  of  directions  for  basic  and  applied  research  and 
development  in  ATR,  with  the  objective  of  providing  DUST  and  DARPA 
with  the  seeds  for  future  initiatives  in  this  area. 

(2)  For  the  second  consecutive  year  Dr.  Mujdat  Cetin  was  asked  by  conference 
organizer,  Mr.  Edmund  Zelnio,  to  participate  in  a  panel  dealing  with  advanced 
radar  imaging  methods,  held  as  part  of  the  SPEE  Aerosense  Symposium. 

(3)  Prof.  Willsky  was  invited  to  participate  in  the  AFOSR-AFRL/IF  Strategic 
Planning  Workshop,  held  at  Dartmouth  College,  August  2002.  The  intent  of 
this  meeting  was  to  define  strategic  directions  for  research  and  development  in 
the  information  sciences,  broadly  defined.  Prof.  Willsky  will  likely  chair  the 
next  of  these  workshops,  tentatively  planned  for  2004. 

(4)  Dr.  Martin  Wainwright  was  invited  by  Dr.  Wendy  Martinez  of  ONR  to 
present  some  of  his  research,  supported  by  this  AFOSR  grant,  at  the  2002 
Army  Statistics  Meeting. 

(5)  Dr.  John  Fisher  was  asked  to  give  an  invited  presentation  at  Lincoln 
Laboratory  on  his  work  on  information-theoretic  methods  for  multisensor 
fusion.  This,  together  with  other  interactions  led  by  Dr.  Cetin,  have  led  to 
increasing  collaborations  and  interactions  between  our  research  team  and 
Lincoln  researchers  and  programs. 

Consultative  and  Advisory  Functions 

We  continue  to  be  actively  engaged  in  a  number  of  activities  relevant  to  the 
research  being  performed  under  our  AFOSR  grant: 

(1)  Prof.  Willsky  has  regularly  acted  as  a  consultant  to  Alphatech,  Inc.  in  a 
number  of  research  projects  including  ones  that  represent  direct  transitions  of 
the  technology  being  developed  under  our  AFOSR  Grant.  These  have,  in  fact, 
accelerated  during  this  past  year,  as  we  are  actively  involved  in  several 
transitions  of  our  work. 

(2)  Prof.  Willsky  recently  completed  his  4-year  term  on  the  Air  Force  Scientific 
Advisory  Board.  During  his  term: 
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a.  He  twice  served  as  a  panel  member  for  the  AF/SAB  S&T  Review  of 
AFRL/SN  and  the  relevant  parts  of  AFOSR  supporting  SN. 

b.  He  twice  served  as  a  panel  member  for  the  AF/SAB  S&T  Review  of 
AFRL/IF  and  the  relevant  parts  of  AFOSR  supporting  IF. 

i.  He  participated  in  three  AF/SAB  summer  studies.  These  each 
involved  extensive  visits  to  AF  and  other  service  organizations, 
and  the  writing  of  portions  of  the  AF/SAB  reports  for  the 
studies  briefed  to  the  Secretary  of  the  Air  Force  and  the  AF 
Chief  of  Staff.  Most  recently,  Prof.  Willsky  participated  on  the 
Information  Integration  and  Management  Panel  for  the 
AF/SAB  study  on  Predictive  Battlespace  Awareness, 
commissioned  directly  by  CSAF.  Prof.  Willsky’s  specific  role 
in  this  study  was  to  examine  and  make  recommendations  on 
S&T  needs  for  fusion  to  support  Predictive  Battlespace 
Awareness  (report  is  written  and  is  currently  under  review  by 
the  AF/SAB  prior  to  release). 

c.  In  addition,  because  of  his  recognized  expertise  in  information 
integration  and  fusion,  Prof.  Willsky  was  asked  to  help  support  one  of 
this  current  year’s  studies — on  Machine-to-machine  ISR  integration. 

d.  Finally,  on  completing  his  tenure  on  the  AF/SAB,  Prof.  Willsky 
received  an  Award  for  Meritorious  Civilian  Service. 

(3)  At  the  request  of  Mr.  E.  Zelnio  of  AFRL/SN,  Prof.  Willsky  participated  as  a 
member  of  an  ad-hoc  panel  helping  Mr.  Zelnio  and  AFRL  with  its  plan  for 
technology  insertion  to  meet  both  short-  and  intermediate-term  objectives 
related  to  the  “Tanks  under  Trees”  initiative  requested  by  the  Air  Force  Chief 
of  Staff  in  response  to  needs  identified  in  Kosovo  operations. 

(4)  Through  contacts  made  on  the  AF/SAB,  Prof.  Willsky  has  been  asked  to  act 
as  an  informal  consultant  to  staff  of  the  National  Reconnaissance  Office.  In 
particular,  Prof.  Willsky  has  been  asked  to  provide  advice  on  future  directions 
for  information  technology  and  fusion. 

Transitions 

The  following  are  the  transitions  of  our  research  that  are  taking  place: 

(1)  Our  multiresolution  SAR  discrimination  algorithms,  most  recently  for  the 
classification  of  nonisotropic  scattering  behavior,  are  being  transitioned  to 
Alphatech  for  inclusion  in  advanced  model-based  ATR  algorithms.  The  point 
of  contact  on  this  is  Dr.  Gil  Ettinger. 

(2)  Our  efficient  methodology  for  multiresolution  mapping  and  data  fusion  have 
been  transitioned  to  Alphatech  as  part  of  an  SBIR  program,  through  NIMA, 
on  fusion  of  multiresolution  and  multipass  data  to  produce  high-fidelity  terrain 
maps.  Alphatech  is  currently  engaged  in  planning  for  a  subsequent  use  of  our 
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methods  for  a  program  through  TEC  (the  Army  Topographic  Engineering 
Center).  The  points  of  contact  are  Mr.  Thomas  Allen  and  Mr.  Laughton 
Stanley. 

(3)  Our  work  on  curve  evolution  methods  are  being  transitioned  to  Alphatech  for 
use  both  in  ATR  applications  and  for  NIH-sponsored  R&D  on  brain  image 
segmentation  for  the  temporal  tracking  of  multiple  sclerosis  lesions  and  for 
MRI  segmentation  of  the  prostate  for  in  order  to  guide  cancer  treatment.  The 
points  of  contact  for  this  work  are  Dr.  Gil  Ettinger,  Dr.  Joel  Douglas,  and  Dr. 
Matt  Antone. 

(4)  Transition  of  our  multiresolution  estimation  and  uncertainty  characterization 
algorithms  to  Alphatech  for  a  DARPA  program  on  uncertainty 
characterization  in  littoral  waters.  The  point  of  contact  for  this  work  is  Dr. 
Eugene  Lavely. 

(5)  Transition  of  our  graphical  estimation  methods  to  Alphatech  for  several 
programs,  including  road  network  estimation  for  DARPA’ s  Dynamic 
Database  (DDB)  Program  and  to  several  higher-level  fusion  algorithms 
sponsored  by  DARPA  and  AFRL/IF.  The  points  of  contact  for  this  work  are 
Dr.  Joel  Douglas,  Dr.  Mark  Luettgen,  and  Dr.  Eric  Jones. 

(6)  Transition  of  our  nonparametric  learning  algorithms  to  Alphatech  for  the 
problem  of  HRR  (high-range  resolution  radar)  feature-aided  tracking  (FAT), 
in  which  the  challenge  is  characterizing  variability  in  HRR  profiles  and 
predicting  what  such  profiles  will  look  like  from  new  viewing  geometries  to 
facilitate  fusion  in  a  FAT  system.  The  points  of  contact  for  this  work  are  Mr. 
Brian  Hodges  and  Mr.  Herb  Landau. 

(7)  Transition  of  our  new  algorithms  for  graphical  optimization  to  several 
Alphatech  programs,  including  new  methods  for  data  association  for 
multitarget  tracking  to  be  used  in  a  number  of  programs  including  the  DARPA 
AMSTE  program  being  run  by  AFRL/IF  and  DARPA’s  DTT  program  (points 
of  contact  Dr.  Robert  Washburn  and  Dr.  Mark  Luettgen)  and  to  the  extraction 
of  “links”  in  complex,  heterogeneous  data  and  construction  of  models  of 
behavior  from  huge  data  repositories  under  DARPA’s  EELD  (Evidence 
Extraction  and  Link  Discovery)  Program  (points  of  contact  Dr.  Eric  Jones  and 
Dr.  Robert  Washburn). 

(8)  Transition  of  our  information-theoretic  methods  for  signal  fusion  to  DARPA’s 
ISP  program  (points  of  contact  Dr.  Mark  Luettgen  and  Dr.  Michael 
Schneider). 
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